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ABSTRACT 

The document presents the executive summary of the 
final report of Project STEEL (Special Teacher Education and 
Evaluation Laboratory), a 3-year study at Indiana University. Project 
STEEL achieved four major goals: (1) development, implementation, and 
evaluation of a microcomputer-based observation system for 
codification, storage, and summarization of special education 
trainees* classroom teaching performances; (2) development, field 
testing, and evaluation of computer literacy training procedures and 
materials for preservice and inservice special education teachers; 
(3) development, implementation, and evaluation of a computer-based 
testing system for assessing teacher knowledge; and (4) development 
and preliminary evaluation of a computer-based information management 
system for storing and retrieving data on special education teachers' 
performances during their preservice training program. The report 
concludes that the microcomputer-based observation and feedback 
system was clearly effective in altering teacher behavior, that the 
coinputer literacy training modules developed are an effective 
instructional package, that the computer-based testing system can 
help college instructors assess students* acquisition of cognitive 
objectives, and that the computer-based information management system 
provides an efficient and secure method for total program evaluation. 
(DB) 
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OVERVIEW 

This report describes developed products, research, and evaluation 
regarding the computer-based Special Teacher Education and Evaluation 
Laboratory (STEEL) at the Center for Innovation in Teaching the 
Handicapped (CITH), School of Education, Indiana University, Blooraington, 
Four major goals were achieved in Project STEEL: 

L Development, implementation, and evaluation of a microcomputer- 
based observation system for codification, storage, and summariEation .5 
special education trainees' classroom teaching performances 
(STEEL/MBOS); 

IL Development, Eeld testing, and evaluation of computer literacy 
training procedures and materials for preservice and inservice special 
education teachers (STEEL/C OLT); 

HL Development, implementation, and evaluation of a computer- 
based testing system for assessing teacher knowledge (STEEL/CBTS); ^^nd 

IV. Development and preliminary evaluation of a computer-based 
information management rystem for storing and retrieving data on special 
education teachers' performances during their preservice training program 
(STEEL/IMS). 

Comprehensive descriptions of each of these major accomplishments 
are provided in four separately bound reports (Volumes I through IV, 
respectively). A fifth separately bound report contains the executive 
summary of Volumes I through IV, and should be read first. 

This document contains the Project STEEL Executive Summary only. 
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EXECUTIVE SUMMARY 



Introduction 

In light of findings from the report, A Nation at Risk , there is need 
for improvement in teacher education practices, including the training of 
special educators. More recently, the Holmes Group has suggested a 
number of reforms in the teacher education process as part of an attempt 
to legitimize the profession of teaching (Murray, 1986). It appears that 
there is general dissatisfaction with the quality of public school 
education, and in turn with the quality of teacher education provided by 
post-secondary institutions. 

At the same time, computer technology has become part of 
education with the advent of relatively inexpensive computers. An obvious 
question arises: Are there practical ways that computar technology can 
be used to increase the effectiveness of teacher training? 

Over a decade ago Semmel and his associates (1976) demonstrated 
that computer-based observation of special education trainees and 
feedback on their performance was effective in increasing targeted 
teaching skiUo two-fold to four-fold (Project CATTS). At that time the 
computer-based observation and feedback was not cost-effective or 
transportable, requiring the utilization of an expensive minicomputer. 
More recently, Semmel and Frick (1982) demonstrated that 
microcomputers could be similarly used at much less cost (Project 
CARTLO). Howe^'er, two problems still remained: the microcomputers 
were not very portable, and there were questions concerning which 
teaching behaviors should be emphasized. 

Results from a three-year study by Rieth (1983) verified the 
importance of academic learning time (ALT) for mildly handicapped 
children and their teachers in special education settings. These results 
were consistent with earlier findings in regular elementary classrooms 
with normal children (e.g.. Fisher, et al., 1978). Frick (1984) found that a 
set of teacher behaviors, referred to as direct instruction, was associated 
with very high levels of mildly handicapped student engagement, whereas 
non-direct instruction was associated with much lower levels of student 
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engc'gement with learning tasks. Rieth (1983) also found that lower levels 
of student engagement were associated with a greater likelihood of poor 
performance on classroom learning tasks. 

Given these previous research and development activities, the need 
for improving the quality of teacher education programs, and the 
increased availability of computer technology. Project STEEL was 
initiated with the major goal of extending the Titilization of that 
technology in the training of preservice special education teachers. This 
included the development of a computer-based observation system, 
training special education teachers in the use of computers, development 
of a computer-based testing system, and development of a computer-based 
information management system. 

Project STEEL Objective #1 . The research cited above, along with 
the commercial availability of truly portable, battery-powered, lap-size 
microcomputers, bec^^me the basis of the first major objective of the 
STEEL Project. We addressed the central question: If preservice special 
education teacher trainees are provided with computer-based 
observational summaries on their use of ALT teaching behaviors during 
field experiences (e.g., practicums), wiU they increase their use of these 
critical teaching behaviors, and will this in turn result in higher levels of 
handicapped student on-task behaviors? 

Project ST EEL Objective #2 . Although the number of computers in 
public schools has increased steadily over the past several years (Becker, 
1986), availability of computers ^er se is clearly not sufficient for 
improving the quality of pubHc school education. Effective software 
integrated with curriculum objectives is necessary, and teachers need to 
know how to use the computer software effectively with their students. 
Teacher computer literacy therefore appears to be essential. Moreover, 
teachers of handicapped students face additional problems when 
incorporating computers in classroom instruction. These additional 
considerations need to be addressed in training special educators in the 
use of computers. Thus, the second major objective of Project STEEL was 
to develop and evaluate computer literacy training materials and 
procedures for special educators. 

Project STE EL Objective #3 . One of the problems cited by the 
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Holmes Group is inadequate teacher knowledge of subject -natter— in 
content areas that they are expected to teach as professionals. If teacher 
education programs are going to do something to improve this state of 
affairs, then they must at least demonstrate that teacher knowledge of 
subjects they are expected to teach is adequate prior to awarding 
baccalaureate or higher degrees. Comprehensive testing cf teacher 
knowledge of subject matter is one means of demonstration. 

It IS true that prospective teachers demonstrate their knowledge of 
subject matter by taking tests as parts of various course requirements. 
However, tests and feedback are seldom given frequently in typical 
college courses. The development., administration, and grading of such 
tests places additional burdens on instructor time, "instructors are 
typicaUy Tiot rewarded externaUy for giving more tests, and indeed may 
view the prospect dispjxagingly as ic is less rewarding than other 
activities. Additional testing time also decreases valuable instructional 
contact time with students-unless, of course, the testing is done outside 
of regr.lar class meeting times. It would appear that computer-based tests 
could be part of the solution to this problem, especially in the 
administration and scoring of tests and in keeping records. 

To address the problem of exploring computer technology for 
administering and grading student tasts and for keeping records, the third 
major objective of Project STEEL was to develop and evaluate a 
computer-based testing system for use by course instructors and special 
education teacher trainees. 

Project STEEL Obiective #4. In an attempt to improve preservice 
teachers' pedagogical skills and knowledge of subject matter, it was 
proposed in objectives 1 and 3 above to utiUze computers in the teacher 
assessment process. These solutions wiU result in a significant amount of 
information on teacher trainees-particularly if coUected over a sustained 
period of time. The amount of evaluative information coUected under 
objectives 1 and 3 goes far beyond the typical listing of courses taken 
and grades received. This creates a need to organize, summarize and 
manage this information if it is to be used effectively in making decisions 
about the quaHty of training received by prospective teachers. To this 
end, the fourth major objective of Project STEEL was to develop a 
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computer-based information management system for use primarily by 
faculty and administrators, and secondarily by students for reports on 
their own individual progress* 

Summary > Improvement in the quality of regular and special 
education in the public schools is generally needed. One long-term 
solution to this problem is to improve teacher education programs so that 
better qualified teachers subsequently enter the public schools. To help 
facilitate this process of improvement, we have developed applications in 
Project STEEL in four major areas of training special education teachers: 
1) computer-based observation and feedback on critical teaching behaviors 
during field-based experiences; 2) computer literacy training; 3) computer- 
based testing on subject matter knowledge; and 4) a computer-based 
information management system for special education teacher training 
programs. These four application areas were developed and evaluated over 
a three-year period during the STEEL project. The following sections 
sumi. arize, respectively, the major project activities and results related 
to the four project objectives. For detailed reporting on developed 
products and evaluations, the reader is referred to Volumes I through IV, 
which are bound separately from this summary report. 
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Objective 1: Microcomputer-Based Observati.on System 

The STEEL Microcomputer-Based Observation System (STEEL/MBOS) 
was developed to provide feedback to preservice special education 
teachers regarding their behavior in field settings— e.g., practicums and 
student teaching. The observational feedback pre /ided in this project 
pertained to academic learning time (ALT), which has been shown in past 
research to be related to student long- term achievement (Rieth, 1983; 
Fisher, et al., 1978). When a preservice teacher was observed during 
classroom instruction by a supervisor from the teacher education program, 
his or her behavior and that of students in the class was codified using a 
modified version of the Academic Learning Time Observation System 
(ALTOS— Frick & Rieth, 1981). The coding was performed directly on a 
portable, battery-operated, lap-size microcomputer (Epson HX-20). 
Immediately after the observation period the supervisors also rated on the 
microcomputer specific instructional and management behaviors using a 
checklist comprised of about 65 items. These data were automatically 
stored on a microcassette at the end of each coding session. The 
supervisors also wrote Eeld notes pertaining to specific events which 
occurred during the coding session. 

When computer feedback was given, a summary printout was 
generated on the spot by the microcomputer. As part of the feedback 
procedure, this printout was given to the preservice teacher along with 
the hand-written field notes. The supervisor then discussed the printout 
results and field notes with the preservice teacher, -indicating strengths 
of his/her performance and suggesting areas in need of improvement. 

During the third year of the project a controlled study with a lagged 
replication was conducted to investigate whether or not the computer- 
based feedback improved ALT (see Volume I). Specifically, we were 
interested in whether preservice teachers increased the percent of time 
spent in active instruction and decreased transition time, and if this did 
occur, whether student engagement rates also increased. 

To execute the study, a group of 26 preservice special education 
teachers at Indiana University wert: randomly divided into two groups (A 
& B). During Phase 1 of the stud' (approximately 8 weeks of baseline), 
both groups were observed using the STEEL/MBOS. Computer feedback 
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was not given during this phase, but as traditionaUy practiced, supervisors 
provided feedback based on field notes taken on student teachers- 
behavior. An analysis of baseUne data indicated no initial statistically 
significant differences between the groups on the percent of time spent 
in active instruction, transition and student task engagement. 

Between Phases 1 and 2 of the study, preservice teachers in Group 
A were provided with a three-hour training session on the importance of 
ALT, the relationship of ALT to student achievement, and the 
interpretation of the computer data summaries. They also were provided 
with a teacher's manual which described specific strategies for increasing 
student engagement and student task success (Frick & Rieth, 1983). Group 
B received an alternative three-hour training session on humanistic 
classroom management skills, based on principles from teacher 
effectiveness training (GordoT^i; 1974). 

During Phase 2 of the study. Group A orcservice teachers received 
computer-based feedback in addition to the field notes. Group B also was 
observed with the STEEL/MBOS, but did not receive computer summaries. 
Grcup B did receive field note feedback as before. 

Phase 2 results indicated that Group A preservice teachers 
significantly increased their percent of time spent in active instruction, 
compared with Group B. Group A teachers also spent significantly less 
time in passive instruction and transition activities compared with Group 
B. Although pupils of Group B preservice teachers significantly increased 
their active on-task behavior during Phase 2 compared with Phase 1, the 
pupils of Group A teachers evidenced higher rates of on-task behavior 
than did those of Group B during Phase 2. 

Between Phases 2 and 3 Group B received the ALT training as had 
Group A earlier. Group A teachers received no further training but were 
given semester summaries of their STEEL/MbOS results. 

During Phase 3 of the study both groups received computer feedback 
summaries as well as field notes. Phase 3 served as a lagged repUcation 
of the Phase 2 results in order to discount threats to external validity 
such as maturation, learning through experience, historical effects, etc. 
When comparing Phase 2 vs. 3 only. Group B preservice teachers showed 
positive changes simila- to those observed earUer in Group A. Group B 
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teachers significantly i:;creased the percent of time spent in active 
instruction nnd significantly decreased the percent of transition time 
during Phase 3. Further, no significant difference was observed for 
student engagement rates for Group B teachers when comparing Phases 2 
and 3. 

Discussion . The results of this study showed clearly that 
observational feedback pertaining to specific instructional behaviors, 
coupled with an explanation of the purpose of such observation, was more 
effective than a traditional f-£ld note feedback approach in increasing 
preservice teaching behaviors related to academic learning time (ALT). 
These results indicate that this procedure holds considerable potential for 
training preservice teachers on a set of critical teaching behaviors 
related to ultimate student achievement. Moreover, the study indicated 
that computer-generated feedback, per se, was not effective in altering 
teacher behavior unless combined with an explanation of a) the purpose of 
the procedure, and b) the meaaing of the specific behavioral categories 
used in the observation system and their implications for pedagogical 
practice. Student acceptance of the program changed dramaticaUy when 
they were provided with specific instructions reL-.ted to the purpose and 
utility of the methodology. Results also indicated that the traditional 
field note approach should not be abandoned, and that a combination of 
computer-based feedback and field note analysis is ideal both in terms of 
increasing teaching effectiveness and in enriching preservice teachers' 
understanding of the complexity of the instructional environment. 

The current results, albeit encouraging, must be weighed in terms of 
a cost-benefit analysis of the total training program. Some of the costs of 
the STEEL/MBOS Involve data storage and transmission, observer training 
and calibration, and acceptance of the procedures by supervisory 
personneL 

First, the STEEL/MBOS entails the coUtction of a considerable 
amount of observational information related to preservice teac'.ier 
performance in field sr igs. This poses problems of data storage and 
analysis. The lap-helc .rocomputers store data on a microcassette. 
These microcassettes must be periodicaUy coUected and information on 
them transmitted to a hrger computer database. Data may be aggregated 
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through various retrievals from the database to provide semester or 
annual summaries on each preservice teacher, as well as summaries across 
all teachers in the program during some period of time. The ktter 
information may be useful in making decisions regarding the overall 
effectiveness of the teacher training program (see Volumes I and IV of 
this report for examples of output from such retrievals). This process 
requires personnel time at each step-data must be transmitted from the 
microcassettes to che larger computer system, and retrievals from the 
larger database need to be perfomed. The expense and time spent on data 
transmission, storage, retrieval and analysis must be weighed against the 
utility of the information for evaluating student performance and overaU 
program effectiveness. 

A second consideration in evaluating the benefits of the 
STEEL/MBOS concerns the cost in time and energy related to observer 
training. Approximately 15 to 20 hours were required to train observers 
to reach an acceptable level of competency for field placement. 
Moreover, an additional four hours per observer were required to 
calibrate skiUs. The cost.', of training and maintaining observers' skills 
must be weighed, especially if a new group of observers must be trained 
anauaUy. A further consideration related to observer training concerns 
the potential stress that some observers may experience when learning 
the procedures and reaching an acceptable criterion. 

Other factors to consider include the perceived inconvenience oJ the 
computers. Despite their compactness, the computer's weight (about seven 
pounds) was burdensome to some observers. Computer batteries must be 
regularly re-charged, and if not done properly, can result in occasional 
losses of data. Further, the noise of the computer-resident printer 
required supervisors to retire to isolated locations to make printout 
summaries in order to avoid disruption of on-going classes. 

A final consideration in evaluating the efficacy of an approach such 
as the STEEL/MBOS concerns the general perception of supervisory staff 
regarding the role of technology in the training program and the 
usefulness of ::he database. In this study the observers were generally 
positive regarding the methodology after the ourpose of the program was 
explained to them and they observed the changes in preservice teacher 
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performance. However, the supervising faculty member viewed the 
methodology as an irrelevant departure from a field note feedback 
procedure which she had previously developed and refined. The additional 
demands on time and inconvenience in managing the methodology were 
seen as prohibitive for this individual. She also remained skeptical of the 
usefulness of the observational information for program evaluation 
compared to traditional information sources. Thus, although the 
methodology proved effective in positively influencing important 
preservice teacher behaviors and maintaining a record of these, 
acc'.ptance of the program by key staff members as a viable alternative 
to traditional teacher training procedures remains a critical factor in its 
potential adoption and implementation in a teacher education program. 
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Obiective 2 . Computer Literacy Training 

A sacont' '-or accomplishment during the first year of the STEEL 
project inv^ .ne development and pilot testing of a computer literacy 
training package (STEEL/COLT) for special education teachers. The 
COLT modules were designed to be integrated in a coUege level teacher 
education course. They c ntain lecture notes for course instructors, 
printed materials for students, microcomputer software, and 
transparencies. In addition, test items are included for assessing learner 
acquisition of knowledge and skills taught. 

In developing the content for the STEEL/COLT modules, one of the 
initial tatks undertaken by the project team involved identification of 
preliminary instructional objectives for the computer Uteracy training 
component. After appropriate major objectives had oeen identified, the 
staff conducted a literature search related to those objectives. The 
search at that time revealed little available research information 
concerning computer education of teachers, and virtually none related to 
the needs of special educators. Thus, much of the -aterial for the 
modules was drawn from the general Uterature, from past research done 
at the Center for Innovation in Teaching the Handicapped (CITH), and 
from tapping the human information resources available within the project 
staff. Eight modules were then developed. The final versions of these 
modules are presented in their entirety in the separately bound Volume n 
of this report. 

Evaluation. To formatively evaluate the STEEL/COLT modules, a 
special education course was offered for credit on the lU Blooniington 
campus during the summer at the beginning the second year of the 
project. A total of 20 students were enroUed in this course, aU of whom 
were graduate level and taught during the school year. Students received 
over 56 hours of direct instruction (two 3 1/2 hour classes each week for 
eight weeks) xn the form of lecture/demonstrations, discussions, and 
hands-on applied experiences. The hands-on experiences constituted 
approximately 30 hours of open-lab, tutorial sessions. Instruction was 
provided by a member of the graduate faculty of the Department of 
Special Education with assistance from CITH's professional staff. 

The two major brands of microcomputers mocdy ]" ly to be found in 
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Indiana schools at that time were the Apple JL and Tandy-Radio Shack 
TRS-80. These were made available in sufficient numbers for use by the 
participants by drawing upon CITH equipment resources and those of the 
School of Education. Liformation was also provided regarding other major 
brands of computers (e.g., IBM PC) to which the students might have 
access, and differences among the various models were discussed when 
appropriate. 

Evaluation of the STEEL/ COLT modules used in the course consisted 
of: 1) an assessment of changes in student attitudes toward 
microcomputers as a result of participating in the course; 2) a 
determination of student mastery of the course content; and 3) a survey 
of students' satisfaction with the training experience. These results are 
discussed in detail in Volume JL of this report, and only will be 
summarized here. 

As a result of participating in the course, individuals apparently 
gained corliaence in their ability to write programs, choose commerciaUy 
available software, and use microcomputers effectively in special 
education classrooms. Their responses further revealed that their 
attitudes regarding the effects of computers on society beca^ie less 
negative. Students also became more critical and less satisfied with the 
instructional value of available software (and rightly so). 

Pre- and posttest versions of a mastery test covering module content 
were administered at the beginning and end of the course in which tha 
STEEL/COLT modules were used. Despite the fact .that the mastery 
posttest was designed to be much more difficult than the pretect, there 
was a highly significant gain in knowledge, indicating increased computer 
literacy. 

Students were highly satisfied with most parts of the course and the 
eight modules. They were especially appreciative of the significant 
amounts of guided hands-on experiences provided. Suggestions for 
improvement of the course and the modules were solicited in addition to 
the above evaluation information. Taken together, these results guided 
Subsequent revisions of the modules. 

During the summer beginning the second year of the STEEL project, 
the course was taught again, using the revised versions of the modules. 

Er|c 1 4 
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SimiUr evaluation activites and results were obtained as described above. 
See Volume E of this report for details. 

During the thiid year of the STEEL project, one module was selected 
for further development in response to teacners' evaluations during the 
previous two years. The SuperPILOT authoring language section (Module 
5) was considerable revised and extended, then was formally tested and 
evaluated once more during an intensive workshop near the end of the 
third year of the project. In total, more than 60 teachers were trained 
over the course of the STEEL project regarding microcomputer 
applications in special education classrooms. 

In summary, the eight STEEL/ COLT modules, when used ir. 
conjunction with a graduate level course in computer Uteracy for special 
education teachers, appeared to be very successful in producing positive 
changes in attitudes toward computers, in helping students master course 
content, and in leaving students feeling satisfied with the course. 

A note of caution for one interested in adopting the ST EEL/ COLT 
modules and who is not highly experienced with using computers in 
education: He or she would be well advised to take such a course first as 
a student before trying to teach it. Alternatively, if one is able to learn 
well alone, it would be advisable to provide at least two semesters of 
lead time to prepare for teaching these materials. 
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Obiective 3: Co .ipoter-Bas ed Testing System 

During the fir?-. y.?r of STEEL, most project resources were 
allocated to development and impk mentation of objectives 1 and 2 as 
described above. Considerable time was spent, however, in specifying the 
design of the computer-based testing system, evaluating and selecting 
hardware and software for development of the testing systen, and 
preliminary e;.periment?tion with kernel routines that the testing system 
would be built upon. 

For the second year of the project. Version 1.0 of the STEEL 
Computer-Based Testing System (STE EL/ CRTS) was developed and 
debugged. During the summer of the beginning of the third year, initial 
pilot testing of the STEEL/CBTS was undertaken with students in the 
STEEL/COLT course and another graduate level course taught by one of 
the principal investigators (Le., students took their mastery tests using 
the STEEL/CBTS). Further pilot testing and revisions of the STEEL/CBTS 
software continued during that year and a research study was conducted 
to test the predictive validity of an adaptive testing strategy that was 
included as part of the testing system. 

Foricative evaluation indicated that the testing system worked 
reliably, as planned, with over 400 successful test administrations to date. 
Security of test item pools and databases for storing test results was 
never violated insofar as could be detected. After initial debugging of a 
rare minor problem, no data on test results have been lost. Response 
times for item presentation and answer judging were excellent in a 
time-sharing environment-even with a relatively large number of heavy 
users. Answer judging was completely reliable for all alternative response 
items (i.e., multiple-choice, true-false), and reliable almost all the time 
for constructed responses (i.e., fill-in-the-blank, short-answer >-the latter 
depending on the Cest developers' answer judging specifications. The 
adaptive testing strategy proved to be highly reliable, while reducing test 
taking time an average of 75 to 80 percent. 

Formative evaluation also revealed the need for revising parts of the 
STEEL/CBTS, adding options not present in Version 1.0 and making the 
STEEL/CBTS more friendly to instructors when developing test item pools 
and retrieving test results. It was not possible, however, in the time 
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frame of the current project to embellish the STEEL/CBTS software. 
Formative evaluation also indicated some hesitancy by faculty to use the 
STEEL/CBTS. The need for a regularly available, easily scheduled, 
supervised testing site also was identified. 

A description of the testing system and results of formative 
evaluation are presented in the separately bound Volume III of this report. 

STEEL/CBTS design considerations. The following parameters were 
considered for the design and subsequent development of the 
STEEL/CBTS: 

1. The testing system was made highly secure from unauthorized 
access, so it can be used for formal evaluation and grading purposes. 

2. The testing system was configured to be highly reliable. Not only 
does it work correctly, but it also does not lose or mix up data. 

3. The testing system was designed for easy access from various^ 
locations on campus, provided that authorized access is permitted. 

4. The testing system was developed to be flexible for ^'nstructor 
creation of a great variety of test item formats. It is posf/.uie and 
relatively easy to develop test items locatable anywhere on the display 
that utilize: a) high-resolution color graphics; b) different sizes, fonts, 
slants and rotations of text; c) instructor-determined answer judging 
criteria regarding spelling, capitalization, punctuation, extra words, word 
order, synonyms, noise words, and alternative correct and incorrect 
responses; and d) answer formats that include constructed responses, as 
well as multiple-choice, true-false, and fill-in- the -blank formats. 

5. The testing system can administer items by a variety of methods— 
e.g., conventional fixed length tests with irems in a predetermined 
sequence, conventional tests with random item selection, and adaptive 
testing strategies— without changing the items themselves . 

6. The testing system was designed so that it can be extended 
relatively easily by competent professional programmers. The source code 
is modularized and documented within. 

7. The testing system was configured to run on computer systems 
commonly found on coUege campuses, ranging from relatively small and 
inexpensive to large and expensive systems— without any modifications in 
the code. 

ERIC '7 
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8. Parts of the testing system are not independently executable, 
unless the context is exactly correct (Le., without other complementary 
parts present and in the "right" places). For example, if someone were 
able to obtain an executable copy of a test item pool, the copy could not 
be successfully run by itself, and would be otherwise unintelligible. 

9. Parts of the testing system can be decentralized in the 
computer's directory and accounting structures, so as to make it 
extremely difficult and time consuming to discover the locations of all 
the pieces— Le., to minimize vulnerability to accidental discovery, 
intentional theft, or sabotage. 

10. The size of the testing system and icem pools is not limited by 
physically available computer memory. A vircual memory computer system 
is therefore required. 

Software and hardware considerations . Given the above design 
specifications for the STEEL/CBTS, software possibilities were next 
investigated. It was concluded that in order to maximize flexibility of 
item developsient by instructors, a good authoring language was needed. 
We chose to go with an authoring language for instructor test item 
development, since this would maximize flexibility and power. We 
expected to develop authoring aids for instructors to minimize their 
learning time (i.e., templates for typical item formats). We also expected, 
wherever possible, to take advantage of existing instructional materials 
and computer-assisted instruction on the language itself and various 
editors required to develop text files, graphics, and character fonts. 

Before settling on a choice of an authoring language, hardware 
possibilities were next investigated. Microcomputers were rejected 
because they typically lack the computing power and size required for the 
resident, centralized testing system. Although some microcomputers can 
be networked and support time-sharing, none that we could find supported 
virtual memory operating svstems or had adequate CPU speed to handle 
more than a few users at once. For security, memory and record keeping, 
the best choice appeared to be minicomputers with virtual memory 
systems, that maintain upward and downward software compatibility 
across various sizes and models, and that are commonly found on coUege 
campuses. 
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One computer system was identified which met all of the above 
criteria: VAX supermini's, mini's, and supermicroco mputers (Digital 
Equipment Corporation). The VAX series aU run compatible versions of 
the VMS operating system and are virtual memory machines; they can be 
flexibly configured depending on anticipated usage. 

After selecting a hardware system, we next considered authoring 
languages that would run on VAX's. Here our choices were restricted at 
that time-either DAL or Producer. Though the Producer was somewhat 
more friendly as an authoring environment, and frequently used for 
in-house development at DEC, its response judging capabilities were 
relatively primitive and unsatisfactory for our purposes. This left the 
Digital Authoring Language (DAL), which is a highly structured lang- ige, 
contains instructions for creating high-resolution, color text and graphics, 
and has very flexible and powerful response judging capabilities. DAL is 
also accompanied by a reasonably good graphics editor and character set 
editor. VMS supports a very easy-to-use text editor (EDIT/EDT), similar to 
a word processor, for developing and modifying source code. Furthermore, 
there is computer-assisted instruction available on DAL, VMS and 
EDIT/EDT. 

A disadvantage of DAL is that aU text/graphics output is coded in 
ReGIS (Remote Graphics Instruction Set). This means that test items 
developed in DAL can only be run on workstations or terminals capable of 
interpreting ReGIS, such as GIGI and VT240 terminals. 

Selection of DAL was not a perfect solution, but, in our opinion, it 
is the best currently available authoring language for the VAX series. We 
beHeve that the advantages of DAL significantly outweigh the 
disadvantages. 

Despite considerable advances in microcomputer technology in the 
last five years, microcomputers were considered infeasible delivery 
systems for the STEEL/CBTS. The major disadvantages of microcomputers 
involve Umitations in networking capabiUties, de-centralization of record 
keeping, and greater problems with security and management of test item 
pools and data on test results. For example, to operate the system, 
multiple disk copies of the same test must be made to test a group 
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simultaneously. Separate data disks need to be maintained for each 
simultaneous user, and then these separately recorded results need to be 
subsequently aggregated into a master file on a bird disk from each of 
the individual floppy disk^. There is considerable room for error in the 
human management of this process. 

Furthermore, the use of microcomputers carries considerable risk in 
loss of data. Microcomputer test takers can defeat the data storage 
process during or after a test by removing data disks during a disk write 
operation, by failing to latch disk drives, or by turning off machines 
during disk-write operations. Lastly, the use of microcomputers would 
restrict the size of the test-item pools due to limitations of physical 
memory in individual microcomputers. 

The problems inherent in the use of " microcomputers for the STEEL 
test program were avoided by using a VAX mainframe environment which 
allowed centralized testing and record keeping, multiple test-.ng site 
options, simultaneous administration, unrestricted item pool and data 
storage, faster test administration and information retrieval, and 
sitDplified and more secure data security and management. 

Development of the STEEL/CBTS. The STEEL/CBTS was developed 
as a set of DAL programs integrated by VMS DCL command files. One of 
the programs (AUTHOR) performs various security checks, and if the 
student passes them, allows the selection of a test from a menu provided 
for a given course and section. If an instructor has authorized access to 
the test chosen at that time, control is next passed to Program TADMIN, 
which administers the test according to criteria and methods set by the 
instructor in advance. Program TADMIN also keeps complete records on 
every item administered during a test, as well as summary records for 
each administration. 

Two utility programs were developed as part of the STEEL/CBTS for 
use by instructors. Program VAUDATE checks the integrity of 
registration files created for courses and sections by instructors. These 
registration files contain student identification information, personal 
passwords, grading criteria, test identification information, methods cf 
administration, test passwords, and date/time slots for when tests can be 
taken. Program DBMSSTEEL allows course instructors to retrieve test 

ERIC '^^^ 



19 



results for individual students or all students registered in a given course 
and section. Txiese results can be output in screen or hardcopy formats. 
Most importantly, the instructor can dump the database into a sequential 
file format suitable for input to almost any DBMS or statistical package. 

Over 11,000 lines of DAL code were written for these four main 
STEEL/CBTS programs. 

Use of the STEEL/CBTS by students . A detailed description of the 
STEEL/CBTS with sample screen displays observed by students is provided 
in Volume IE of this report. To access the system a student sits at a 
terminal and logs onto the VAX computer where the testing systv m 
resides. Next, she or he enters identification, course, and section 
numbers. If registered in the testing system for that course/section, the 
student must next enter his or her personal password to continue. If the 
password is correct, a menu of tests is displayed. Having chosen a test 
and if authorized to take it at that time, the student must then enter the 
current test password. 

When the student passes all these security checkpoints, the chosen 
test is begun. After a welcoming message, general test directions are 
presented on how to answer questions, how to correct typing mistakes, 
and how the test will be administered. Test items are then presented one 
at a time. At the end of the test the student is informed of the number 
of correct and incorrect answers and the decision outcome associated 
with that score (e.g., pass/fail, a letter grade, etc.). The student can 
optionally review his or her answers to items missed on the test, but 
cannot review test items themselves. FinaUy, a parting message is 
displayed and the student is logged off the computer. 

Summary. Considerable field testing of the STEEL/CBTS indicated 
that it did work reliably. No data were lost during formal testing, and the 
security system worked satisfactorily. Most of the important evaluation 
results from field tests were discussed earUer and further detail is 
provided in Volume lEL While a number of embellishments could be made 
in the STEEL/CBTS to make it more convenient for users, cessation of 
the project precluded these. Further funding may be sought to support 
refinement and expansion of the system. 
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Objecdv e 4: Computer-Based Information Management System 

The discussion of objectives 1 and 3 indicated that significant 
amounts of assessment information can be obtained on prese.rvice special 
education teachers concerning their demonstration of pedagogical skills in 
field experience settings and knowledge of subject matter relevant to 
their teacher education program. Thus, there was a need to organize, 
summarize and manage this information for purposes of decision making 
regarding preservice teacher proficiencies. 

To this end, the computer-based STEEL Information Management 
System (STEEL/IMS) was developed. The STEEL/IMS was designed for 
storage and retrieval of observational data, test results, student 
background information, and other data pertaining to student progress in 
the special education program. The primary use of the STEEL/IMS was 
intended for faculty and administrators in advising students, tracking 
student progress, and in making recommendations concerning graduation 
and subsequent job appUcations by teachers. The secondary use of the 
STEEL/IMS was intended for preservice teachers in the program, so that 
they could, upon proper identification, access their personal records in 
the database. 

During the first year of the project, design parameters were 
specified for the STEEL/IMS, and hard ware /soft ware characteristics were 
considered. During the second year, a prototype system was developed 
using an information management system caUed the Scientific Information 
Retrieval Database Management System (SIR DBMS). Tliis prototype was 
used to store and retrieve observational data coUected on teacher 
performance during field experiences and teacher background information. 
During the third year, the STEEL/IMS was revised and further extended 
by adding the capability of storing and retrieving test data obtained from 
the STEEL/CBTS, and by adding an electronic mail system for 
communication among faculty and students in the special education 
program. 

The STEEL/IMS was designed to provide menu-driven on-Une help at 
almost every response point in the program. Volume IV of this report 
provides a detailed description of the software and sample screen displays 
for typical user sessions. 
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Formative evaluation of the STEEL/IMS indicated that the software 
performed remarkably well. Input of data records, storage, and a variety 
of retrievals were accomplished quickly and without error. The security 
system of the STEEL/IMS is very sophisticated and permits different 
levels of access to cpecific records, depending on the security 
authorization level of a user. Each user is availed to various levels of the 
database through personal passwords. This security system was easy to do 
in SIR DBMS, since these features are built into the software system. The 
STEEL/IMS was extensively formatively evaluated and subsequently 
revised during in-house testing. It was completed and ready for use by 
faculty near the end of the last year of the project. Unfortunately, 
because of the extensive time spent in development and limited faculty 
time, little information regarding user interface issues could be collected 
before the project terminated. To maintain and update the STEEL/IMS a 
person must be responsible for managing the system who has some 
familiarity with the SIR DBMS language. Clerical assistance is also 
needed for data input-e.g., transmitting computer-based observational 
data from the microcomputer to the STEEL/IMS, entering student 
background information, and entering information on results from tests 
not administered by the computer-based testing system. 

Design considerations. Many of the same design principles that were 
used in developing the computer-based testing system described earUer 
were also considered for the STEEL/IMS. The major exception to this is, 
of course, difference in the purpose of the IMS. We concluded, as we did 
for the testing system, that security, reliabiUty, centralization of record 
keeping, and ease of access was maximized if the STEEL/IMS resided on a 
single computer. Thus, the VAX environment, which supports a 
time-sharing system and is networked to workstations available at many 
different cempus locations, was selected to support the STEEL/IMS. 

Hardware/ software considerations . Microcomputers were not totally 
discounted as resident machines for the STEEL/IMS, primarily because at 
any given time there would only be a few simultaneous users of the 
STEEL/IMS, urlike the situation during group testing. However, a very 
powerful micro would be required to operate the system, with at least 
one-half megabyte of internal memory and 20 to 40 megabytes of 
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secondary storage on a hard disk, with a high-speed tape backup system. 
These conditions would also be easily met by typical minicomputer and 
mainframe systems. 

Use of the STEEL/IMS. Detailed descriptions and examples of screen 
displays and retrieval outputs are provided in Volume IV of this report. 
Only a brief description is provided here. To access the STEEL/IMS, a 
user first logs onto the computer and then starts the program. Second, 
she or he is presented with a brief welcoming message and disked if an 
introduction is wanted (for first-time users). If so, a brief explanation of 
the STEEL/Ii:s is provided, and conventions are explained on usage (e.g., 
how to exit some process, get help, correct typographical errors, etc.). 
Third, the user is queried about the kind of terminal or workstation being 
used, so that screen output can be tailored to that device. Database 
security clearance is then checked. The user is asked for his/her 
STEEL/IMS user name and password. This prevents unauthorized access 
and also determines the security level clearance for how the system can 
be used. For example, a faculty member will see one kind of menu of 
functions that can be performed in the STEEL/IMS, whereas a student 
would see another menu. 

Only someone with system manager privilege would see the most 
extensive menu which permits actual modification of the database (see 
examples of screen displays in Volume IV). If accessed by a faculty 
membar, for example, the STEEL/IMS functions would include the 
following: 

1. Access the mail service. 

13. Report the student profiles. 

31. Summarize a student's observation data. 

32. Summarize a student's checklist data. 

33. Summarize each student's observation data. 

34. Summarize each student's checklist data. 

35. Summarize all student observation data. 

36. Summarize all student checklist data. 

41. Report a student's test data (by course/section/objective). 

42. Report each student's test data (by course/section/objective). 

43. Analyze test items (by course/section/objective). 
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1£ the STEEL/IMS were accessed by someone with system manager 
privileges, the menu incudes aU of the above and many more functions on 
managing the database (e.g.. purging student profiles, adding observation 
data, storing the database on magnetic tape, printing the access log. 
etc.). 

Each STEEL/IMS retrieval can be delimited by date ranges and other 
criteria. For example, a faculty member may want to retrieve an 
observational data summary on student X during his senior year only. Or, 
he or she might want criterion-referenced test (CRT) results on aU 
students in a course or section that were administered during the faU 
semester. Depending on how extensive the output from a retrieval is, it 
can be viewed on the terminal display or routed to a nearby high-speld 
printer for permanent hardcopy. 

m summary, the STEEL/IM3 permits a variety of ways of retrieving 
information on students in the special education program. Security is 
excellent, and database functions are limited by the type of security 
clearance one has been assigned. 
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Summary 

The central goal of Project STEEL was to utilize computer 
technology in ways that could facilitate the process of training special 
education teachers. If teachers are better prepared by their training 
programs, we would expect nhat they would do a better job of teaching in 
the public schools than is now occurring. 

Over a three-year period Project STEEL staff developed, 
implemented and evaluated technological applications for use in special 
teacher education programs. Four major objectives were accomplished: 1) 
the Microcomputer-Based Observation System for pre /iding feedback to 
preservice teachers during field experiences, 2) Computer Literacy 
Training for special educators consisting of eight modules for use in a 
coUege level course, 3) the Computer-Based Testing System for assessing 
preservice teacher knowledge of subject matter and 4) the Information 
Management System for storing and retrieving information on students in 
the teacher education program. 

In project STEEL, we feel we have advanced the field by creating a 
prototypical computer-based system which facilitates preservic? and 
inservice teacher training. The microcomputer-based observation and 
feedback system developed on this project was clearly effective in 
altering teacher behavior in training environments. The computer-base j 
testing system offers coUege instructors a powerful aide for assessing 
students' acquisition of cognitive objectives in their courses and for 
providing them feedback on their progress. The computer-based 
information management system provides an efficient and secure method 
for toTal program evaluation. Finally, the computer literacy training 
modules offer an effective instructional package for training special 
educators to use microcomputer technology in special education 
classrooms for mildly-handicapped youngsters. 
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